I spent some time investigating what was wrong, and managed to fix the broken audio and artifacts in all combinations of sampling rate, mono and stereo alike. It seems to be working fine to me, now (even disabling/enabling denoising on the fly, where you can actually hear it at work), so I marked the PR as ready for review. That said, I'll obviously wait for more feedback before (if) we merge this, also taking into account the jitter buffer PR would need to be merged first (which means I'll have to rebase this accordingly).

Hi! Is there a chance that this feature will be in VideoRoom Plugin?

Hi! Is there a chance that this feature will be in VideoRoom Plugin?

No, because the VideoRoom plugin only relays packets, it doesn't decode them as the AudioBridge does.

Hi! Thanks for this PR!

We were using a frontend version of the same RNNoise with the default model via WASM and it doesn't affect the original voice quality much. (https://github.com/jitsi/rnnoise-wasm)

But, when testing this PR it results in a very "strangled" voice quality compared to the WASM one.

Would you say that this is related to the fact that it's used on the original lossless microphone samples before encoding frontend side, or rather something related to the usage of RNNoise in this PR ?

PS: I could provide 3 simultaneous audio recordings of the original, denoised-frontend, denoised-janus sound in a noisy environment if it helps

Thanks again !

Would you say that this is related to the fact that it's used on the original lossless microphone samples before encoding frontend side, or rather something related to the usage of RNNoise in this PR ?

I think it depends on the sampling rate you're using in the AudioBridge. If you use, e.g., 16000, resampling will be performed, and I think it may be that the way we do it causes what you hear. If in a 48000 room the audio is good, and comparable in your tests, that's likely it.

Indeed i retested with 24000/48000 room sampling rate and it works perfectly fine. I'll be moving this to production tonight for some 500 daily users (some OPUS/48000 some PCMA/8000 (SIP & PSTN bridged from the SIP plugin to the AudioBridge plugin)) I'll let you guys know how it works out for quality & CPU usage. Thanks again great work 🙏

We've seen no significant CPU usage increase with this enabled in production (2 to 4 percentage points increase in average CPU usage across multiple 32 CPU VMs). Although, it does seem to add some "barely noticeable" cumulative delay in audio contributions with denoise enabled (just under the 200ms mark) for hour long conferences.

@Odinvt are you sure the observed delay is due to the denoiser?
We recently merged the jitter buffer in the master branch (now merged in this PR) and that might also explain a bit of latency.
To double-check this, use the Admin API for an handle with the delay and read the values of buffer-in and queue-in.

@atoppi Indeed I was on v1.1.4 stable before. I only switched to v1.2.0 because this merge was rebased on top of speexdsp's jitter buffer. So i never got to actually test v1.2.0 separately. I will pay attention to the mentioned values for both denoise enabled and disabled, and get back to you asap. Thanks !

Hi, we are testing your PR. Denoise is working when initially joined with denoise: true. But changes on the fly with new configure request aren't working:

{"body":{"denoise":false,"request":"configure"},"handle_id":8708560238099099,"janus":"message","session_id":6601772272643216,"transaction":"QUrR+WT5jq4"}
{"body":{"denoise":true,"request":"configure"},"handle_id":8708560238099099,"janus":"message","session_id":6601772272643216,"transaction":"+6ZA40wJa3g"}

requests abobe doesn't change anything on the fly. Is anything wrong with them?
I also tried with ice restart, sending new jsep and denoise flag, but it does no effect.

@lminiero any plans to merge this?

requests abobe doesn't change anything on the fly

@spscream Does it work when using the synchronous denoise_enable and denoise_disable requests instead?

@brave44 we do plan to merge this, but not sure yet, as we're still waiting for more feedback; besides, we may have to improve how it works when doing 8000/16000 sampling rates.

@brave44 we do plan to merge this, but not sure yet, as we're still waiting for more feedback; besides, we may have to improve how it works when doing 8000/16000 sampling rates.

Thanks, yeah, for us it works fine with 4800 sampling rate.

@Odinvt @spscream @brave44 the algorithm has been totally refactored, could you please test the last revision?

@Odinvt are you sure the observed delay is due to the denoiser? We recently merged the jitter buffer in the master branch (now merged in this PR) and that might also explain a bit of latency. To double-check this, use the Admin API for an handle with the delay and read the values of buffer-in and queue-in.

I apologize for the late reply. The audio delay was due to our "tree" scaling's forwards going through a VXLAN over VPN which encryption was not hardware accelerated.
We've been running this branch with the jitter buffer rebase since September with no issues with 24000/48000 sampling rates in production after successful stress tests.

requests abobe doesn't change anything on the fly

@spscream Does it work when using the synchronous denoise_enable and denoise_disable requests instead?

@lminiero the synchronous requests work fine we've been using them in production also.

@Odinvt @spscream @brave44 the algorithm has been totally refactored, could you please test the last revision?

@atoppi Will do. Thanks !

@atoppi thanks, we will check it

Can't help on forks, sorry. If you can find if any commit we made is causing the problem, please do let us know.

@lminiero sure, we will do our best.
But fork isn't related to ab, only vr logic affected(some custom fields added to participants state).

@lminiero we checked on master + our commit today and we have no troubles on it. We will check the latest changes on current branch afternoon.

@lminiero moved to latest changes from this branch, no problems now for us.
We will monitor it until tomorrow, looks like everything is fine.

We have some troubles with audiobridge. If publisher have bad network connection we hear crack sounds and it appears on every call with such clients. After change ec52fbb is a bit better, but doesn't remove issue completly. How can we debug it and help to address it?

btw on vr publisher from the same users we don't have these cracks

I tested it using clamsy utility https://jagt.github.io/clumsy/ - cracks starts after adding drops over 5%

Unless these reports are related to the RNNoise effort, this is the wrong place to talk about it.

Unless these reports are related to the RNNoise effort, this is the wrong place to talk about it.

I created #3297 issue to adress it.

We are currently testing your PR. As part of our testing process, we're setting up an audio bridge room with the RNNoise feature activated, by setting the "denoise" option to true.

In certain circumstances, we've noticed an echo effect. This typically happens when using the laptop's built-in speaker and microphone. Interestingly, this issue does not occur when headphones are utilized.

@mail2mhossain that sounds more like an issue with the client environment.

This typically happens when using the laptop's built-in speaker and microphone. Interestingly, this issue does not occur when headphones are utilized

Do your clients have echo cancellation mechanisms?
Typically browsers do and it should be enabled by default if the device supports it.
A quick one-liner test in a browser is the following:

(await navigator.mediaDevices.getUserMedia({ audio: true})).getAudioTracks()[0].getSettings().echoCancellation

More importantly, is this indeed a problem with the RNNoise integration? Meaning, does it happen when RRNoise is enabled in the AudioBridge, but doesn't when the AudioBridge doesn't? If not, it's irrelevant to this PR, and as Alessandro said a client side problem (just wear a headset).

It appears that the issue we're encountering, specifically the echo experienced when using the laptop's built-in speaker and microphone, may indeed be related to the absence of RNNoise integration. We are considering integrating RNNoise as a potential solution to address this challenge.

The issue occurs both when RRNoise is activated in the AudioBridge and also when it's not integrated with the AudioBridge.

Our application utilizes a desktop client that incorporates the Sipsorcery WebRTC library, and for audio functionalities, we're leveraging the SIPSorceryMedia.SDL2 library. Based on your feedback, it appears that echo cancellation features are expected to be handled by the client library, in this case, SIPSorceryMedia.SDL2. Consequently, I'm understanding that RNNoise might not support this specific type of noise cancellation, correct?

Could RNNoise effectively resolve the echo issue we're facing with the laptop's built-in speaker and microphone? Or, would it be more advisable to incorporate a noise reduction library on the client side?

Could RNNoise effectively resolve the echo issue we're facing with the laptop's built-in speaker and microphone? Or, would it be more advisable to incorporate a noise reduction library on the client side?

Noise suppression and echo suppression are very different processes.

Noise suppression subtracts from a generic stream what is being recognized as background noise and could be performed in both media server (audiobridge with RNNoise in this case) or in the endpoint.
On the other hand echo suppression is a process happening on an endpoint that subtracts from the microphone stream the echo that is bouncing back from the loudpseakers.

Merging.